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ABSTRACT 

This document reports on three focus groups comprised of 
state officials, local practitioners and supporters, and researchers who were 
convened to provide input on strategies for assessing and validating the 
effects of tech prep. Part I provides a brief summary of the groups* 
discussions, including major points and broad themes in these four topic 
areas identified in a literature review: outcomes, measurement, how to define 
a tech prep student, and other design and data collection issues. The summary 
reports the following: (1) most agreed a national tech prep evaluation should 

focus on student outcomes but noted the program cannot impact students 
without first changing educational institutions; (2) a national evaluation 
should also consider these outcomes: participation and retention in secondary 
programs, transition to postsecondary ’ education, and participation and 
retention in postsecondary programs; (3) measurement concerns included 
difficulty in tracking, delayed transition to postsecondary education, 
employer perspective in selecting student outcomes, and difficulty in 
defining a "program-related" career; (4) many believed a consensus definition 
for tech prep students that states could refine was reasonable; and (5) other 
design and data collection issues were the need to include input and 
contextual variables, how to coordinate a national evaluation with Perkins 
III requirements, relationship to state and local efforts, and state and 
local needs for technical assistance. Part II contains reports for each of 
the three focus groups, organized around the same four topics. (YLB) 
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from the original document. 
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Introduction 

Federal funding for tech prep programs began in 1990, under Title HIE of the Carl D. 
Perkins Vocational and Applied Technology Education Act Amendments (Perkins II, P.L. 101- 
392). The federal appropriation for tech prep programs has grown from $63,000,000 in Fiscal 
Year (FY) 1991 to $106,000,000 in FY 2000. Yet relatively few data are available to document 
the effects of tech prep programs on students. Such documentation is of critical importance to 
policymakers responsible for decisions about the design, implementation, and funding of tech 
prep programs, especially in light of new legislative requirements concerning accountability. To 
obtain this information, the U.S. Department of Education’s (ED) Office of Vocational and Adult 
Education (OVAE) is preparing for a future impact study of the federal tech prep program. 

OVAE is also interested in identifying the research and technical assistance needs of tech prep 
programs. 

Research Triangle Institute (RTI) is supporting OVAE’s efforts by: 

• Building a framework for future tech prep research and evaluation; and 

• Exploring future directions for program practice, in order to inform ED’s 
leadership and technical assistance efforts. 

To accomplish these goals, we undertook several activities, including: 

• A review of the literature on tech prep, including the current status of tech prep 
programs and a comparison of approaches to evaluating tech prep programs and 
their outcomes. 

• Conduct of focus groups comprising state officials, local practitioners and 
supporters, and researchers, who will supplement information obtained through 
the literature review and provide input on strategies for assessing and validating 
the effects of tech prep; 

• Preparation of a Focus Group Report that reviews the key issues identified by 
respondents; and 

• Preparation of a “Framework for Future Tech Prep Research and Evaluation ” 
and a comprehensive briefing. The research and evaluation framework will 
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review evidence on the effects of tech prep and provide recommendations 
regarding: (1) approaches to more rigorously assessing the student and program 
outcomes associated with the program; and (2) future directions for program 
practice, including suggestions for efficiently disseminating successful evaluation 
strategies and indicators. We will also present this information at a briefing for 
ED officials. 

The first product of the study documented the results of our review of the literature on 
tech prep.‘ This document indicated that decisions would need to be made about several key 
issues before national, state, and local evaluations of the tech prep program could focus 
successfully on student outcomes. The literature review served as the basis for the focus groups’ 
discussions, which are described in this document. 

Selection of Focus Group Participants 

ED s plans for the task order called for the focus groups to comprise a variety of 
individuals who were knowledgeable about issues pertaining to the evaluation of tech prep 
programs at the national, state, and local levels, including: 

• State directors of vocational and technical education; 

• State tech prep coordinators; 

• Representatives of relevant national organizations; 

• Researchers responsible for national- and state-level studies of the program; and 

• Representatives of local tech prep consortia. 

We selected focus group participants based on input from OVAE officials and findings 
from the literature review; nominees were subject to the approval of ED. While the individuals 
who attended the three sessions came from many parts of the country, the small number of 
sessions (and the limited size of each focus group) made it impossible to include a representative 
from each state. Instead, we selected participants because of their expertise in the evaluation of 



'Elliott, B. G„ and Statelman, T. M. (2000). Tech prep: Building a framework for future research, 
evaluation, and program practice. Literature review. Revised draft. Research Triangle Park NC' Research 
Triangle Institute. 
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tech prep programs, including individuals who represented a variety of perspectives (i.e., local, 
state, and national). Consequently, with the exception of those who were themselves state 
administrators, peirticipants’ views do not necessarily represent those of vocational education or 
tech prep officials in their states. 

Appendix A provides a complete list of focus group members, along with their 
affiliations. As shown in the appendix, a number of OVAE representatives also attended the 
session, primarily as observers. 



Conduct of the Focus Groups 

We convened the focus groups in Washington, DC, on March 29-30, 2000. Each of the 
three sessions lasted for one-half day, with the issues identified in the literature review serving as 
an organizing framework for the discussions. Those issues included: 



The outcomes that tech prep programs should produce. The level of flexibility 
that federal legislation affords to states and local consortia allows tech prep 
programs to address a wide variety of goals and objectives. It also complicates 
the task of evaluators, who will need to make choices about the student outcomes 
that the study should examine. 

Measurement issues surrounding selected outcomes. Researchers will also need 
to consider the feasibility of collecting data for particular outcomes of interest. 

For example, while entry into program-related employment after completion of an 
cirticulated secondary-postsecondary course of study will be an outcome of 
interest, some programs may not have been in existence long enough for students 
to achieve this outcome. 

How to define a tech prep student. Such decisions are critical to the conduct of 
an evaluation that accurately reflects the impact of the tech prep program. 
Understanding “who counts” as a tech prep student is the basis for examining 
differential effects of such programs under varying state and local conditions. 

Ways to address other design and data collection issues, including the need to 
include input and contextual variables (e.g., student characteristics, stage of 
implementation, local labor market conditions, and resources), coordination with 
Perkins III requirements and state and local evaluation activities, and ways to 
address data collection issues. Under this topic, group members also provided 
input concerning the types of technical assistance that would enable states and 
local programs to collect student outcome data. 
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To serve as a springboard for the groups’ discussions, we provided members with: 

• A list of possible outcomes suggested by Perkins HI (see Exhibit 1 ). This exhibit 
was prepared by RTI for discussion purposes only. 

• One definition of a tech prep student, developed by the National Association for 
Tech Prep Leadership (NATPL) on the basis of information furnished by about 20 
states. NATPL provided this information for discussion purposes only, and has 
not endorsed the definition, which is shown in Exhibit 2. 

Organization of This Document 

This document is organized into two parts: 



Part I provides a brief summary of the groups’ discussions, including major points 
and broad themes in each of the four topic areas (outcomes, measurement, 
defining a tech prep student, and other design and data collection issues). 

Part II contains reports for each of the three focus groups, organized around the 
same four topics. 
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Exhibit 1 

Outcomes Suggested by Perkins III 



Legislative requirement 


Outcome 


Combine minimum of two years secondary 
and two years postsecondary education 


• Completion of articulated 

secondary/postsecondary program 


Integrate instruction and utilize work-based 
or worksite learning 


• Achievement of employability skills/work 
readiness 


Provide technical preparation in a career 
field 


• Placement in program-related career 

• Retention of, and advancement in, 
employment in program-related career 


Build competence in academic areas and 
workplace skills 


• Improvements in academic skills 

• Improvements in workplace skills 


Lead to associate degree, baccalaureate 
degree, or postsecondary certificate in a 
career field 


• Attainment of associate degree in career 
field 

• Attainment of baccalaureate degree in 
career field 

• Attainment of postsecondary certificate in 
career field 

• Attainment of postsecondary credentials in 
career field 


Lead to placement in appropriate 
employment or to further education (Section 
202(a)(3) 


• Placement in program-related career 

• Transition to baccalaureate program in 
career field 

• Transition to additional postsecondary 
training in career field 



9 

er|c 



5 



Introduction 



Exhibit 2 

One Definition of a Tech Prep Student 



Tech prep secondary student 

Has indicated an intent to pursue, and is enrolled in courses within, a recognized tech prep 
education plan that consists, at a minimum, of two years of secondary and two years of 
postsecondary study; is carried out under a written articulation agreement; may allow the student 
to earn postsecondary credit while in secondary school; and leads to a specific postsecondary 
two-year certificate, degree, technical diploma, or apprenticeship. 



Tech prep postsecondary student 

Has participated in the secondary portion of a recognized tech prep education plan that consists 
at a minimum, of two years of secondary and two years of postsecondary study; is carried out 
under a wntten articulation a^eement; and leads to a postsecondaiy two-year certificate, degree, 
technical diploma, or apprenticeship and has enrolled in the postsecondaiy portion of that 

education plan. The student may have transferred in college credit earned in the secondary 
school. ■’ 



Source: National Association for Tech Prep Leadership, March, 2000. 
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f 1: Summary of Focus Group Discussions 

Summary of Focus Group Discussions 



Exhibit 3 shows some of the key issues in the design of a national evaluation, as 
identified by focus group participants. It also lists their recommendations for addressing these 
issues. The balance of this section provides a brief summary of participants’ comments 
concerning: 



The outcomes a national evaluation should examine, and their suggestions for 
additions to those shown in Exhibit 1; 

• Measurement issues surrounding specific outcomes; 

• The definition of a tech prep student shown in Exhibit 2, and the implications of 
evaluators’ decisions about a definition; and 

• Other design and data collection issues. 

Only major points from the discussions of individual groups, or themes that became apparent in 
; more than one session, are presented here. The individual focus group reports in Part H provide 
complete details on each section. 

Outcomes 



While most participants agreed that a national tech prep evaluation should focus on 
student outcomes, they also noted that the program cannot impact students without first changing 
educational institutions; e.g., it must produce changes in staff development opportunities, 
instruction, guidance and counseling, and involvement of businesses, industries, and labor 
unions. Because these changes appear before student outcomes, it would be useful to identify 
institutional outcomes and to link those outcomes with particular effects on students. For 
example, an institutional outcome would be the presence of a comprehensive career development 
system; for students, the result of such a system would be an improved awareness of career 
options. Because states and local consortia place varying degrees of emphasis on particular 
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Part 1: Summary of Focus Group Discussions 



Exhibit 3 

Key Issues in the Design of a National Evaluation 
and Focus Groups’ Recommendations for Addressing Them 



Issue 


Focus Group Recommendations 
for Addressing Issues 


The tech prep program cannot impact students 
without first changing educational 
institutions. 


Identify institutional outcomes and link 
those outcomes with particular effects on 
students. 


The program is designed to permit multiple 
“exit points. ” 


Include a continuum of outcomes in the 
evaluation design. 


Evaluation requirements should align with the 
Perkins III core indicators. 


Study outcomes for tech prep participants, 
concentrators, and completers at both the 
secondary and postsecondary levels. 


An evaluation could identify tech prep 
students either on the basis of intent or 
retrospectively (i.e., after they complete the 
secondary portion of the program). Each 
approach has advantages and disadvantages. 


Carefully consider the implications of 
decisions about how to identify tech prep 
students. 


If the evaluation adopts a retrospective 
definition, students will be identified in such 
a way that they are automatically likely to be 
the most successful academically (i.e., those 
who complete the program are likely to have 
higher academic ability than those who did 
not). 


Identify appropriate comparison groups and 
develop ways to control for differences in 
academic ability. 


It is difficult to interpret outcome data 
without contextual information. 


Use aggregate state- and consortium-level 
data to provide contextual information, or 
collect contextual information in a random 
sample of consortia within each state. 


Because states and local programs place 
varying degrees of emphasis on particular 
elements of the program, they will not all be 
equally successful in producing specific 
outcomes. 


Collect information concerning local 
implementation of the program. 



(continued) 



Part 1: Summary of Focus Group Discussions 



Exhibit 3 

Key Issues in the Design of a National Evaluation 
and Focus Groups’ Recommendations for Addressing Them (continued) 



Issue 


Focus Groups Recommendations 
for Addressing Issues 


Ideally, a national evaluation should provide 
both student outcome data and contextual 
information that administrators and 
practitioners can use for program 
improvement. 


Include a supplemental “in-depth ” study at 
a sample of sites. 


There are several discrepancies between the 
Perkins III core indicators and the outcomes 
suggested by the tech prep legislation (see 
Exhibit 1). 


Align data collection requirements for 
Perkins HI and the national evaluation to 
the extent possible, considering ways in 
which evaluation data could supplement data 
from the core indicators. 


States are still in the process of developing 
data collection systems to comply with 
Perkins HI requirements. 


Decide whether OVAE should allow time 
for states to improve their data collection 
systems before conducting a national 
evaluation. 


In anticipation of a national evaluation, some 
states and local programs might postpone 
their own evaluation activities. 


Design a national evaluation to build on, 
rather than duplicate, state efforts. 
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Part 1: Summary of Focus Group Discussions 



elements of the program, they will not all be equally successful in producing specific student 
outcomes. 

Perhaps, one participant suggested, it would be useful to think about a continuum of 
outcomes for tech prep students. Among the earliest outcomes might be retention in secondary 
school, attainment of state-defined academic skills, and receipt of a high school diploma. Later, 
a participating student might enter postsecondary education (possibly with advanced credit, and 
hopefully without the need for remediation). Finally, a tech prep student would complete an 
articulated two-year postsecondary program, and perhaps enter a baccalaureate program. 

Group members suggested that, in addition to examining the outcomes shown in 
Exhibit 1, a national evaluation should also consider: 

• Participation and retention in secondary programs. Retention in secondary 
school is an extremely important outcome to parents and legislators. 

• Transition to postsecondary education, particularly without the need for 
remediation. Tech prep’s emphasis on transition to postsecondary education, 
group members noted repeatedly, is a unique feature that distinguishes the 
program from other forms of vocational education. Inclusion of this outcome 
would enable evaluators to document the program’s effect on students who enter 
postsecondary education but do not complete an articulated program; a group that 
may nevertheless benefit from participation in tech prep. 

• Participation and retention in postsecondary programs. Since most community 
colleges have open entry policies, it is also important to examine student 
persistence in postsecondary education. 

They also identified two measures for inclusion in the study design: 

• Attainment of state-established academic skills as a measure of achievement of 
academic skills. Local principals and superintendents are extremely interested in 
tech prep’s role in improving student test scores. 

• Earnings as a measure of retention and advancement in program-related careers. 
Congress has made it clear that it not only expects vocational programs to 
improve academic achievement and college-going rates, but also to increase 
earnings. Consequently, a national tech prep evaluation should collect 
information on students’ initial and later earnings. The program’s impact on 
earnings may become apparent while students are still in college, since tech prep 
students are often able to obtain high-paying part-time jobs before they complete 
their studies. 
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1‘ Summary of Focus Group Discussions 

Measurement 

Focus group members identified a number of measurement issues surrounding individual 
outcomes. Exhibit 4 summarizes their concerns about each outcome, and shows possible 
strategies for addressing these concerns, as suggested by the groups’ discussions. It also 
identifies the individual focus group reports that provide full details about specific points. 

Defining a Tech Prep Student 

Many participants believed that it was reasonable to talk about developing a consensus 
definition for tech prep students; states could refine this definition so long as the one they 
employed contained, at its core, the consensus definition. Most also believed that the definitions 
currently used by their own states were reasonably consistent with the one developed by NATPL. 
However, they also expressed concerns about particular components of that definition, which are 
outlined in Exhibit 5. The exhibit also shows possible strategies for addressing each concern, as 
suggested by the groups’ discussions. 

Group members suggested that a national tech prep evaluation must either: 

(1) identify tech prep students based on their intent to pursue a tech prep program; or (2) wait 
until individuals have completed the secondary portion of the program to determine who is a tech 
prep student (i.e., retrospectively). This decision has important implications for the design of the 
study. Identifying students on the basis of intent would allow evaluators to administer 
assessments as students progress through high school. However, since students’ intentions 
frequently change over time, such a definition might hold states and local programs accountable 
for individuals who leave the tech prep program in order to pursue other options. If, on the other 
hand, an evaluation identifies students retrospectively, it will be more difficult for researchers to 
collect information about students’ performance in secondary school. Further, since those with 
the highest academic ability would be most likely to complete the program, evaluators would 
need to control for academic ability when identifying comparison groups. 
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Part 1: Summary of Focus Group Discussions 



Exhibit 4 

Measurement Concerns 
and Possibie Strategies for Addressing Them* 



Outcome 


Measurement Concerns 


Possible Strategies for 
Addressing Concerns 


For Details, 
see Report 
for Group 
Number(s): 


Outcomes Shown in Exhibit 1 






Completion of 
articulated secondary/ 
postsecondary program 


• In some states, tracking 
students from the secondary 
to the postsecondary level 
can be quite difficult. 

• Students do not always 
make the transition to 
postsecondary education 
immediately after high 
school. 

• Some institutions will not 
release data to help states 
and local programs track 
students because of 
concerns about compliance 
with the Family Educational 
Rights and Privacy Act 
(FERPA). 


• Develop software that links 
secondary and postsecondary 
records. 

• Offer financial incentives to 
local consortia that monitor 
progress between levels. 

• Track students for several 
years after high school 
graduation. 

• Obtain waivers from students 
and their parents to address 
institutions’ FERPA 
concerns. 


1,3 


Achievement of 
enployability 
skills/work readiness 


• It is important to consider 
the perspective of employers 
in selecting student 
outcomes, especially in 
identifying employability 
skills. 

• Not all states assess 
achievement of 
employability skills. 

Among those that do, the 
definition of “employability 
skills” may vary. 


• As new skill standards and 
certification processes are 
developed, consider using 
them to help document 
attainment of this outcome. 

• Measure employer 
satisfaction by surveying a 
national sample of employers 
to assess students’ 
achievement of employability 
skills. 


1,2 



* As suggested by the focus groups' discussions. (continued) 
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Part I: Summary of Focus Group Discussions 



Exhibit 4 

Measurement Concerns 

and Possibie Strategies for Addressing Them* (continued) 



Outcome 


Measurement Concerns 


Possible Strategies for 
Addressing Concerns 


For Details, 
see Report 
for Group 
Number(s): | 


Placement, retention, 
and advancement in 
program-related careers 


• It can be difficult to define 
what counts as a “program- 
related” career. 

• Students may achieve 
desirable outcomes even if 
they do not complete 
articulated programs or 
enter program-related 
careers. 

• Itmaynotbetothe 
program’s advantage to link 
it too narrowly with specific 
occupations. 


• Use the Classification of 
Instructional Programs (CIP), 
Higher Education General 
Information Survey (HEGIS), 
or Occupational Employment 
Statistics (OES) system to 
categorize secondary and 
postsecondary courses. 

• Study placement in both 
program-related careers and 
unrelated careers without 
making a judgment that one is 
better than the other. 


1,2.3 1 


Improvements in 
academic skills 


• This outcome should 

measure achievement, rather 
than improvement, of 
academic skills. 


• Use state assessment data, 
which are of great interest to 
state and local administrators, 
to measure student 
achievement of state academic 
standards. 

• Consider administering a 12**’ 
grade assessment of academic 
skills, since much tech prep 
activity occurs in the 11**’ and 
12*** grades. 

• Employ Grade Point Average 
(GPA) as a measure of 
academic skills. 


1,3 1 


Improvements in 
workplace skills 


• This is an important 
outcome, but one that is 
difficult to measure. 


• Use employer surveys to 
measure attainment of this 
outcome. 


3 1 


Attainment of associate 
degree, baccalaureate 
degree, postsecondary 
certificate, or 
postsecondary 
credentials in career 
field 






■ 



* As suggested by the focus groups’ discussions. (continued) 
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Part 1: Summary of Focus Group Discussions 



Exhibit 4 

Measurement Concerns 

and Possible Strategies for Addressing Them* (continued) 



Outcome 


Measurement Concerns 


Possible Strategies for 
Addressing Concerns 


For Details, 
see Report 
for Group 
Number(s): 


Transition to 
baccalaureate program 
or additional 
postsecondary training 
in career field 






1 


Additional Outcomes Identified by Focus Groups 


Institutional outcomes 
that produce particular 
effects on students 


• Because states and local 
consortia place varying 
degrees of emphasis on 
particular elements of the 
program, they will not all be 
equally successful in 
producing specific 
institutional or student 
outcomes. 


• Document the contribution of 
tech prep to school change, 
rather than trying to show 
how program funds (which 
account for only a small 
portion of school budgets) 
affect student outcomes. 




Participation and 
retention in secondary 
programs, including 
both tech prep and 
other programs 






1 


Transition to 
postsecondary 
education, particularly 
without the need for 
remediation 


See “Completion of articulated 

secondary/postsecondary 

program.” 




1.2,3 


Persistence in 
postsecondary 
programs, including 
both tech prep and 
other programs 






.2 



* As suggested by the focus groups' discussions. 
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Exhibit 5 

Concerns About Particuiar Components of the NATPL Definition and 
Possibie Strategies for Addressing Them* 



Component 


Focus Group Concerns 


Possible Strategies for 
Addressing Concerns 


For Details, 
see Report 
for Group 
Number(s): 


Intent to pursue 


• Even where there is a statewide 
definition, some local programs 
may not adopt it. 

• Not all state definitions depend 
on intent; those that do may 
define the term in various ways. 

• Since students’ intentions 
frequently change over time, 
identifying students on the 
basis of intent might hold states 
and local programs accountable 
for individuals who leave the 
tech prep program in order to 
pursue other options. 


• In states that employ written 
education plans, these documents 
will help to identify tech prep 
students. 

• School officials who conduct 
annual reviews of students’ 
education plans will identify 
individuals who leave the tech 
prep program. 

• One alternative to identifying 
students on the basis of intent is 
to classify those who take 
particular courses, or 
combinations of courses, as tech 
prep students. 

• Another alternative is to identify 
students retrospectively: i.t., 
after they complete the secondary 
portion of the tech prep 
program.^ 


1,2,3 


Enrolled in 
courses within a 
recognized tech 
prep education 
plan 


• Not all states have recognized 
education plans. 

• The word “recognized” could 
raise concerns about whether 
the plan is to be approved at the 
local, consortium, or state level. 

• “Enrolled” may have a different 
meaning in various states (e.g., 
one day, completion of two 
courses). 


• Allow states to develop their own 
definitions of what counts as a 
“recognized tech prep education 
plan.” 

• States may need guidance about 
what education plans should 
include. . 

• States will have an incentive to 
define “enrolled” carefully, in 
order to avoid being held 
accountable for individuals who 
do not participate fully in, or 
complete, the program. 

• In order to align with Perkins III,, 
it might be desirable to identify 
tech prep participants, 
concentrators, and completers. 


1,2 



* As suggested by the focus groups' discussions. (continued) 



^However, this approach would make it more difficult for evaluators to collect information’on students’ 
performance in secondary school. Since students with the highest academic ability would be most likely to 
complete the tech prep program, it would probably be inappropriate for a national evaluation. 



er|c 




Part 1: Summary of Focus Group Discussions 



Exhibit 5 

Concerns About Particular Components of the NATPL Definition and 
Possible Strategies for Addressing Them* (continued) 



Component 


Focus Group Concerns 


Possible Strategies for 
Addressing Concerns 


For Details, 
see Report 
for Group 
Number(s): 


Carried out 
under a written 
articulation 
agreement 


• Not all articulation agreements 
pertain to tech prep programs. 

• The definition might need to 
refer to program, rather than 
course, agreements. 

• The term “articulation 
agreement” could refer to either 
a broad statewide agreement or 
a specialized local initiative. 


• Allow states to define what is 
meant by a “written articulation 
agreement” (e.g., program vs. 
course agreements, broad 
statewide or specialized local 
initiative). 


1,3 


May allow the 
student to earn 
postsecondary 
credit while in 
secondary school 
(secondary 
definition) OR 
The student may 
have transferred 
in college credit 
earned in the 
secondary school 
(postsecondary 
definition) 


• These statements should be 
deleted because they do not add 
to the definition. 




1,2.3 


Leads to a 
specific 
postsecondary 
two-year 
certificate, 
degree, technical 
diploma, or 
apprenticeship 


• Wording does not correspond 
exactly to that of Perkins III, 
which refers to an associate 
degree, postsecondary 
certificate, or baccalaureate 
degree. 


• Modify wording to correspond to 
that of Perkins III. Also add 
apprenticeship. 

• Add employment to the 
postsecondary definition. 


1 



* As suggested by the focus groups' discussions. 
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other Design and Data Collection Issues 

Under this topic, focus group members discussed several issues, including: (I) the need to 
include input and contextual variables in the design of a national evaluation; (2) how a national 
evaluation should be coordinated with Perkins III requirements; (3) its relationship to state and 
local efforts; and (4) state and local needs for technical assistance. 

Input and Contextual Variables 

Ideally, participants suggested, a national tech prep evaluation should provide both 
student outcome data to support funding requests at the federal, state, and local levels, and 
programmatic information that administrators and practitioners can use for program 
improvement. To accomplish the latter purpose, its design might include a supplemental “in- 
depth” study at a sample of sites. 

Focus group members identified a number of input and contextual variables that might be 
relevant, including: 

• The ways in which discontinuation of federal funding for school-to-work 
programs affects tech prep programs. 

• The level of state and local funding for tech prep programs. 

• The statewide context within which tech prep programs operate; e.g., whether the 
state has its own legislation governing tech prep, and where within state 
government the program is housed. 

• Issues concerned with leadership; e.g., the level of support that tech prep 
receives from the state director of vocational and technical education and how 
activities funded by the basic state grant and tech prep are coordinated. 

Coordination with Perkins III Requirements 

As an OVAE representative pointed out, there are several discrepancies between the 
Perkins HI core indicators and the outcomes shown in Exhibit 1.^ States may use a variety of 
measures to comply with the Perkins III requirements; a national evaluation, however, could 



^For a complete listing, please see Exhibit 6 in Part II. 
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employ a single measure. Perhaps more importantly, it would permit researchers to supplement 
the core indicator data with additional information that might be more pertinent to the tech prep 
program. For example, instead of relying on state assessment data, tech prep evaluators might 
decide to administer an assessment of academic skills at the 12• **' grade level (since most state 
assessments are administered at the 10**’ grade level, while many tech prep activities do not take 
place until the last two years of high school). 

Some states have indicated to OVAE that they will not be able to provide the 
performance data required by Perkins III for several years. Consequently, Department 
representatives wondered whether it would be wise to wait until 2002 to undertake a national 
tech prep evaluation, in order to give states time to improve their data collection systems. 



Coordination with State and Local Efforts 

Federal officials should educate state officials about the importance of conducting their 
own tech prep evaluations. ED should also indicate that a national study will build on, and not 
replicate, state efforts, and encourage states to share evaluation data with federal administrators 
responsible for tech prep. 

Technical Assistance Needs 

States would benefit from technical assistance to help them collect comparable data for 
both tech prep and other vocational education students. OVAE might also help states and local 
programs address the confidentiality concerns created by the Family Educational Rights and 
Privacy Act (FERPA). 



Conclusions 

Focus group participants provided a wide range of ideas and recommendations 
concerning; 

• Methods for addressing key issues in the design of a national evaluation, which 

include — 
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*■ The importance of recognizing the connection between implementation 
of particular program components and specific student outcomes; 

*■ The need to include a continuum of outcomes in the evaluation design; 

► The desirability of aligning evaluation requirements with the Perkins III 
core indicators; 

► The implications of evaluators’ decisions about how to identify tech prep 
students; 

► The importance of contextual information for interpretation of outcome 
data; 

► Appropriate timing for a national tech prep evaluation; and 

r The importance of building on, rather than replicating, state evaluation 
efforts. 

• The outcomes that a national evaluation should examine. 

• Measurement issues surrounding specific outcomes. 

• Ways to define a tech prep student, and the implications of evaluators’ decisions 
for the study. 

• Other design and data collection issues. 

Their suggestions should be helpful not only to researchers responsible for a national study, but 
also to evaluators and administrators at the state and local levels, who struggle with many of the 
same issues as they work to evaluate and improve their own tech prep programs. 

Based on input from the literature review and the focus groups, RTI researchers will 
develop a Framework for Future Tech Prep Research and Evaluation that provides 
recommendations regarding: (1) the design of a future evaluation of tech prep; (2) ways in which 
research can support program implementation; and (3) OVAE’s technical assistance activities. 

This document will assist ED in preparing for a future impact study of the program, which will 
provide information of critical importance to policymakers responsible for decisions about the 
design, implementation, and funding of tech prep programs. 
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Group 1 

Group 1 included one state director of vocational technical education and one state tech 
prep coordinator. Two participants were selected for their expertise in state-level evaluation of 
tech prep programs, while one represented a national organization. The remainder were staff of 
local tech prep consortia. This group was concerned primarily with: 

• The addition of several outcomes, including retention in secondary school and 
transition to postsecondary education; 

J ’ 

• The importance of state assessment data to a national tech prep evaluation; 

• Issues that tech prep’s connection with education reform raises for evaluators; 

• The importance of contextual information for interpretation of outcome data. 



Outcomes 



Participants identified several additional student outcomes, including: 

• Dropout prevention, which parents ^d legislators consider extremely important. 
To incorporate this outcome, we could modify the first bullet in Exhibit 1 to refer 
to participation and retention in secondary programs. 

• Transition to postsecondary education, especially without the need for 
remediation. The link to postsecondary education is a feature that distinguishes 
tech prep from other forms of vocational education, although the transition does 
not always occur immediately: some participants reported that their states track 
students for several years after high school graduation. Some individuals, 
however, may not enter college until they are in their 20s. 

• Qualitative outcomes; e.g., changes in attitudes toward learning. 

Most group members agreed that any national tech prep evaluation should include state 
assessment data. They recommended, however, that the outcome of “improvements in academic 
skills” be restated to refer to meeting or exceeding state academic standards. This outcome, like 
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dropout prevention, is very important to parents and legislators; state assessment results are also 
useful for program improvement purposes. One state, Texas, has been able to show that its tech 
prep students have a higher passing rate on the state competency test (the Texas Assessment of 
Academic Skills, which students take at the end of grade 10). The individual responsible for the 
state’s evaluation activities noted, however, that analysts are not able to identify specific aspects 
of the program that account for this success: in theory, it is even possible that higher-achieving 
students are more likely to enter tech prep programs. The state does not attempt to draw 
conclusions about the reasons for the difference in test scores; it simply reports the results. 

Group members questioned whether OVAE intended to examine only student outcomes. 
An OVAE representative indicated that the department did not want to exclude other types of 
outcomes from discussion, although it would be necessary to prioritize outcomes in designing an 
evaluation. Participants suggested that it might be useful to study the effect of the program on 
the entire school, rather than just on tech prep students. Many aspects of tech prep have the 
potential to impact all students: these include increased availability of staff development 
opportunities, integration of academic and vocational instruction, and expanded preparatory 
services at both the secondary and postsecondary levels. Because tech prep funding accounts for 
only a small portion of school and college budgets, it might make more sense to consider hOw 
those dollars contribute to school change over time than it does to try to show the funding’s 
effect on student outcorhes. If ED decided to broaden the scope of the study, it should be 
possible to present results in a way that educates Congress about the program’s broader impact. 
Measuring changes in the school environment, however, can be difficult. Further, because 
schools that do not offer tech prep programs may also participate in staff and curriculum 
development activities, it might be difficult or impossible to identify a comparison group of 
schools. 

Students are more likely to succeed in tech prep programs if they receive appropriate 
counseling to help determine whether the program is right for them. Consequently, one 
participant recommended, a national evaluation should examine the issue of whether they are 
receiving the necessary guidance. This is also an area in which states and local programs might 
benefit from technical assistance. 

One group member suggested that, especially since multiple exit points may help in 
marketing the program, it might be useful to think about a “continuum” of outcomes. Ideally, a 
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tech prep student should earn an associate’s degree. But, before obtaining a degree, he must 
enter a postsecondary program (perhaps with advanced credit, or at least without a need for 
remediation); prior to that, he would receive a high school diploma. Other early outcomes might 
include retention in secondary school, improvements in academic achievement, and changes in 
course-taking patterns. 

Participants also discussed the connection between tech prep and education reform which, 
although positive, raises concerns for evaluators. Where the program is conceived as a school 
reform activity, it may not be possible to identify “tech prep” students or to specify a comparison 
group (except, perhaps, dropouts). Although schools without tech prep programs might provide 
a comparison group, they too may have benefitted from staff and curriculum development 
activities. The link with education reform can also affect the identification of outcomes: one 
participant noted that, in his state, the program emphasizes improved academics rather than 
career-related goals. 

One group member was particularly concerned about the difficulty of studying the 
program s impact on a limited subset of students. In her state, she explained, administrators are 
interested primarily in tech prep s impact on the school as a whole, and on school restructuring 
(e.g., reductions in school size and changes in student counseling practices). Where curricula for 
all students are similar (as is the case for this state’s math instruction), identification of tech prep 
students may be difficult. 

Participants suggested that, in keeping with the Perkins core indicator framework, OVAE 
might wish to study outcomes for tech prep participants, concentrators, and completers at both 
the secondary and postsecondary levels. We might find, for example, that outcomes for the 
casual participant are not as good as those for the concentrator, which in turn might not be as 
good as those for the secondary completer; those for the postsecondary completer might be best. 
The definition of “participant,” however, may depend upon the requirements of state graduation 
plans. 

Finally, one participant offered a suggestion pertaining to the outcome of “completing an 
articulated secondary/postsecondary program.” Her state, she reported, examines outcomes for 
both tech prep students who enter articulated postsecondary programs and those who enter other 
postsecondary programs. 
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Measurement 

In this section, we present the group’s concerns about measurement issues surrounding 
particular outcomes, following the order in which those outcomes appear in Exhibit 1 . We also 
discuss general themes concerning measurement that emerged from the focus group’s 
discussions. 

Measurement Issues for Particular Outcomes 

Completion of articulated secondary/postsecondary program. The feasibility of 
collecting data for outcomes related to both the secondary and postsecondary levels will depend 
to some extent upon the type of data collection system that the individual state has constructed. 
Some states have a system that links secondary and postsecondary data for all schools and 
colleges; e.g., North Carolina is developing software that utilizes electronic high school 
transcripts and identifies tech prep completers through a computerized analysis of course-taking 
patterns. In Illinois, on the other hand — although the postsecondary data system identifies tech 
prep students — secondary school records are on paper. To identify tech prep participants who 
enter college, secondary personnel must manually enter data for individual students. 

Achievement of employability skills/work readiness. Group members expressed 
considerable concern about the feasibility of measuring achievement of employability skills/work 
readiness. While some states (e.g., Illinois) have statewide tests of workplace readiness skills, 
others do not. Some administrators might argue that a student has demonstrated achievement of 
employability skills by completing a course that meets state curriculum standards, and that there 
is no need for additional testing. Further, the definition of “employability skills’’ will vary from 
one state to another, and perhaps even among consortia within a state. 

The political support of employers, one participant noted, is critical for tech prep 
programs. Consequently, it is important to consider their perspectives in selecting outcomes and 
measures: input from employers would be essential in identifying employability skills. 
Measurement of this outcome may become more feasible in the future with the development of 
skill standards and certification processes. 
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Measuring employer satisfaction might be another way to monitor achievement of 
employability skills/work readiness. Some participants reported that local consortia in their 
states already surveyed employers, although response rates were sometimes low. Perhaps, group 
members suggested, employer satisfaction could be measured through a survey of a national 
sample of employers. 

Placement in program-related career and retention of, and advancement in, 
employment in program-related career. Society may have a legitimate interest in knowing 
whether tech prep students enter and advance in “program-related” careers: for example, state 
officials may need to know how well tech prep programs address workforce development needs. 
Nevertheless, most group members agreed, measurement of these outcomes may be problematic 
because of the difficulty involved in determining what counts as a “program-related” career. 
Although the Classification of Instructional Programs (Cff) provides a way to categorize 
secondary and postsecondary courses, it is difficult to link those codes with particular programs 
and occupations. These difficulties are illustrated by the problems that Texas, which uses 
Unemployment Insurance (UI) wage data to monitor students’ employment and earnings, has 
encountered. As a result, participants suggesting deleting the phrase “program-related” in all 
outcomes. Instead, it might be preferable to say “in a career field,” in order to correspond with 
the legislation. It would still be necessary, however, to define what counts as a “career,” rather 
than a job. 

At the secondary level, analysis of course-taking patterns may be required in order to 
identify a student’s “program.” At the postsecondary level, it can be difficult to determine what 
kind of employment counts as “program-related.” One participant reported that students from his 
community college, which offers 60 majors, enter between 800 and 900 different occupations. 
Sometimes, students may use the skills they acquire in school in seemingly unrelated careers. 

Linking tech prep programs with narrowly defined occupations may not be in the 
program’s best interest. Since students are advised that tech prep programs will prepare them for 
a variety of careers, it may not be appropriate to evaluate the program by their entry into a single 
occupation. Instead, students may achieve desirable outcomes in a variety of ways. For 
example, some may raise their expectations and enter four-year programs; others who were at 
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risk of dropping out may become interested in posfsecondary education, but enroll in 
unarticulated programs. 

Participants agreed that employment outcomes were of interest only if they followed 
postsecondary education. Employment directly from high school should count as a successful 
outcome only if it is in a recognized apprenticeship. 

Improvements in academic skills. This outcome should measure attainment, rather than 
improvement, of academic skills, which implies pre- and post-testing (unless evaluators are 
considering student-level aggregates). 

Transition to baccalaureate program in career field and transition to additional 
postsecondary training in career field. Participants wondered if we should clarify whether 
these two outcomes refer to transition to a baccalaureate program or additional training only after 
completion of a two-year program, or whether they also include transition directly from high 
school. One group member noted that her state does permit students to go directly from high 
school to a four-year program, partially as a marketing tool, although it does not promote this 
option. 

Changes in attitudes toward learning. Although it would be interesting to know 
whether participation in tech prep programs causes students to change their attitudes toward 
learning, measuring progress toward this outcome would require the collection of survey data for 
a random sample of students. These data would be expensive and time-consuming to obtain; 
further, since they would be based on self-report, their validity might be questionable. Perhaps, 
one respondent suggested, states could consider monitoring this outcome through their own 
evaluation activities. In fact, researchers responsible for a national evaluation might be able to 
recommend additional ways in which state activities could complement the national effort. 

General Themes 

The following general themes concerning measurement also emerged from the focus 
group's discussions. 

Timing of a national evaluation. OVAE representatives noted that most states chose to 
submit transitional, rather than five-year, plans under Perkins III; some indicated that they would 
not have performance data for several years. They wondered whether it would be wise to wait 
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until 2002 to undertake a national tech prep evaluation, in order to give states time to improve 
their data collection systems. 

Data collection issues. One participant reported that, in her state, some districts do not 
receive any tech prep funds, although they do participate in staff development. Especially since 
no district staff member is assigned to the tech prep program, these districts may be reluctant to 
provide evaluation data. The state \vill have to address this issue, hovv^ever, in order to meet 
Perkins III requirements. 

Identification of students. In some states, students do not realize that they are enrolled 
in tech prep programs. Tech prep, one participant explained, is “invisible to them; an educator’s 
term, not a kid term.” In Maryland, for example, students may see themselves only as having 
met two sets of graduation requirements, in preparation for either a college or career. Some 
administrators may emphasize transition to postsecondary education rather than participation in a 
specific program. In other cases, school officials prefer not to “label” students in order to avoid 
concerns about tracking. 

School outcomes. As noted earlier, tech prep programs may impact the whole school by 
increasing the availability of staff development opportunities, integrated instruction, and 
preparatory services. An evaluation could monitor these changes, although data might not be 
comparable among districts, consortia, or states. For example, a staff development activity might 
be a one-day workshop in some areas, while it is a monthly meeting in others. Further, these data 
would probably not be useful for program improvement purposes. Information on career 
development interventions will come from OVAE’s two-year project with the National Research 
Center for Career and Technical Education, in which Oregon State University will examine the 
impact of various strategies. The study will provide a comprehensive taxonomy of career 
development interventions. 



Defining a Tech Prep Student 

Asked whether they thought it was reasonable to talk about developing a single definition 
for tech prep students, some members of the group expressed the belief that it was. If ED is to 
conduct an evaluation, it must establish such a definition in order to be able to collect outcome 
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data. States can refine the definition if necessary, so long as the resulting definitions do not 
conflict with the consensus definition. 

While most participants believed that the definition of a tech prep student developed by 
NATPL (shown in Exhibit 2) would work in their states, they mentioned a number of concerns 
about particular aspects. In this section, we describe those concerns. 

Intent to Pursue 

Not all state definitions depend on student “intent”; those that do may define the term in 
various ways. For example: 

• New York asks students to sign a registration form, and counts them as tech prep 
students after they complete two technical courses. 

• Illinois counts high school students as tech prep students if they have a written 
career plan indicating that they are preparing to enter what the state defines as a 
tech prep occupation: i.e., one that (1) has an associate’s degree, a two-year 
certificate, or a two-year apprenticeship following high school completion as the 
predominant method of entering the occupation; (2) has opportunities for above- 
average entry wages and career advancement; and (3) requires advanced technical 
skills. 

As a group member from the latter state remarked, the phrase “intent to pursue” indicates that the 
student has received counseling and understands that he is preparing for entry into an career that 
meet the state’s definition of a tech prep occupation. This participant suggested that it might be 
useful to define not only a tech prep student, but also a tech prep program and a tech prep 
occupation. 

Texas — which has a statewide approval process for tech prep programs — does not include 
“intent” in its definition, but simply counts individuals as tech prep students when they enroll in 
the first of a sequence of career/technology courses approved as tech prep. In this state, which 
uses a 4+2 model, individuals can be counted as tech prep students as early as grade 9, and may 
be enrolled under any of the state’s three graduation plans. In their reports, evaluators compare 
the performance of three groups, including: (1) tech prep students; (2) students participating in 
career/technology courses that are not state-approved as tech prep; and (3) all other students. 
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Another concern related to inclusion of “intent to pursue” in the definition is that 
students’ intentions change over time. Unless school officials review students’ plans each year, 
they may not be able to accurately identify tech prep students. 

Enrolled in Courses Within a Recognized Tech Prep Education Plan 

Most participants indicated that their states employed some type of written education 
plan, which might be called a Career Development Plan, Education Learning Plan, Individualized 
Education Plan, or other type of plan. Some expressed concerns, however, about the definition 
of a “recognized tech prep education plan.” 

The word “recognized” could raise concerns about whether the plan is to be approved at 
the local, consortium, or state level. The real issue, participants agreed, is whether or not 
students are participating in a planned sequence of courses. As long as the definition of a 
recognized plan is left to the individual states, this is probably an acceptable term. Since not all 
states currently have recognized education plans, they may need guidance about what such plans 
should include. One participant suggested that they should incorporate the philosophy of 
“Breaking Ranks,” which calls for each secondary school student to have an individualized 
learning plan. 

States also make different decisions about the point at which a student is considered to be 
“enrolled” in a tech prep program. Some may define them as tech prep students on the first day 
of their enrollment under a written plan. In New York, students sign registration forms in the S* 
grade; their plans are reviewed annually. In Texas, which uses a 4+2 model, high school students 
may be classified as tech prep in any of grades 9 through 12, so long as they participate in a 
single course within an approved tech prep program. Illinois, however, does not designate 
students as tech prep until the 11“’ grade. Participants observed that states will have an incentive 
to define tech prep students carefully, so that they will not be held accountable for individuals 
who do not participate fully in, or complete, the program. They also noted that in order to align 
with Perkins III, it might be desirable to identify tech prep participants, concentrators, and 
completers. 

34 
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. Carried Out Under a Written Articulation Agreement 

One focus group member noted that some high schools in her state participated in local 
articulation agreements that did not pertain to tech prep programs; further, not all articulated 
programs are tech prep programs. Another suggested that it might be desirable to refer to 
program, rather than course, agreements; the group, however, agreed to maintain the term 
“written articulation agreement” and allow each state to determine what that means. 

May Allow the Student to Earn Postsecondary Credit While In Secondary School 

(secondary definition) 

The Student May Have Transferred In College Credit Earned In the Secondary 
School (postsecondary definition) 

These statements do not necessarily add to the definition; without them, it would still 
define a tech prep program. A community college representative noted, however, that advanced 
credit is often the most practical approach to eliminating duplicative courses for individual 
students. The majority of community college students no longer enroll directly after high school. 
Therefore, it is difficult to set up courses to accommodate only those who do. 

Leads to a Specific Postsecondary Two-year Certificate, Degree, Technical 
Diploma, or Apprenticeship 

This wording does not correspond exactly to that of Perkins III. Participants suggested 
using the exact legislative wording, which refers to an associate degree, postsecondary certificate, 
or baccalaureate degree; Perkins II also referred to apprenticeship. They also suggested adding 
employment to the postsecondary definition, since enabling students to hold high-paying jobs 
while enrolled in postsecondary education would be a desirable outcome. 

Other Design and Data Collection Issues 

Under this topic, participants discussed several issues, including: (1) the need to include 
input and contextual variables in the design of a national evaluation; (2) how a national 
evaluation should be coordinated with state and local efforts; and (3) state and local needs for 
technical assistance. 
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Input and Contextual Variables 

An OVAE representative explained that, while the next national evaluation will not be 
restricted entirely to examining student outcomes, descriptive work will be secondary. 

Participants noted, however, that it is difficult to interpret outcome data without contextual 
information. For example, it may not be reasonable to expect students in states with high 
unemployment rates to achieve the same outcomes as those in states with low unemployment; or, 
those in areas where employment is primarily in agriculture to earn as much as others who have 
access to jobs in high-tech industries. Similarly, programs that benefit from high levels of state 
funding might produce better outcomes. To determine what input and contextual variables 
should be included, it would be useful to know how evaluation data will be used at the federal 
level; e.g., whether they will be employed in funding decisions. 

One way to obtain contextual information would be to use aggregate state- and 
consortium-level data, which some — but not all — states could provide. Even this information 
does not necessarily “tell the whole story”; for example, it does not reflect local labor market 
conditions or how much emphasis individual schools place on contextual teaching and career 
guidance. Local coordinators, however, cannot be expected to collect such information. 

Qualitative studies, like Debra Bragg’s current work, provide information that helps to 
inform state and local evaluation practices, and that is useful for program improvement. Perhaps, 
participants suggested, a national evaluation could collect contextual information in a random 
sample of tech prep consortia within each state. Another approach would be to gather these data 
at a small number of selected sites through a supplemental “in-depth” study. 



Coordination with State and Local Efforts 

To coordinate state and local evaluation activities, one group member recommended that 
OVAE could find out more about the data that states already collect on a regular basis through: 

• A survey of states; or 

. • A review of state plans, which describe accountability systems. 

These activities would give ED a better idea of what data elements it could expect to collect, at 
what expense. Such a survey, or review, might be consistent with OVAE’s work in developing 
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the Perkins IH core indicator framework, which includes identifying common measurement 
approaches and definitions. 

Technical Assistance Needs 

P^icipants’ suggestions concerning technical assistance needs pertained primarily to the 
image of the tech prep program. Local programs, one suggested, desperately need high-quality 
public service announcements to which they can attach their own identifying information. To 
improve the program’s image, tech prep should be linked with other initiatives such as ED’s New 
American High Schools and Small Learning Communities, as well as those of High Schools That 
Work. 
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Group 2 

Group 2 included one state director of vocational technical education and one state tech 
prep coordinator, along with two researchers and one representative of a national organization. 
The remaining participants were staff from local tech prep consortia. Like the individuals in the 
first focus group, Group 2 recommended the addition of transition to postsecondary education as 
an outcome. Participants also discussed: 

• The possibility of identifying intermediate institutional outcomes for each 
legislative requirement, and linking those outcomes with particular effects on 
students; 

• Difficulties involved in monitoring students’ placement, retention, and 
advancement in program-related careers; 

• Advantages and disadvantages of identifying tech prep students on the basis of 
intent or retrospectively (i.e., after they complete the secondary portion of the 
program; and 

• Issues concerning coordination with Perkins III requirements. 



Outcomes 



Participants identified one additional student outcome: 

• Transition to and persistence in postsecondary education. Tech prep’s emphasis 

on transition to postsecondary education is a unique feature that distinguishes it 
from other forms of vocational education. Since most community colleges have 
open entry policies, it is also important to examine student persistence in 
postsecondary education. 



While they agreed that a national evaluation of tech prep should examine student 
outcomes, group members also pointed out the importance of understanding the ways in which 
the program produces those outcomes, and the reasons vyhy some states and consortia are more 
successful than others. Before tech prep can impact students, it must first bring about changes in 
educational institutions; e.g., changes in guidance and counseling, instruction, staff development. 
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and involvement of businesses, industries, and labor unions. These changes take time to occur 
(particularly when the school is implementing a comprehensive program), but will become 
apparent before student outcomes. Consequently, it might be useful to identify intermediate 
institutional outcomes for each legislative requirement and to link those outcomes with particular 
effects on students. For example, an intermediate institutional outcome would be the presence of 
a comprehensive career development system; for students, the result of such a system would be 
an improved awareness of career options. 

For each element of the program, an evaluation should examine corresponding student 
outcomes. However, because states and local consortia place varying degrees of emphasis on 
particular elements of the program, they will not all be equally successful in producing specific 
outcomes. A more thorough understanding of local implementation would help to inform our 
interpretation of outcome data. 

Tech prep programs, one participant noted, prepare students for a wide variety of high- 
tech and “low-tech” jobs; some emphasize specific occupations, while others target clusters of 
careers. Evaluators should consider whether all of these programs can be expected to produce 
the same outcomes. 

Measurement 

In this section, we present the group’s concerns about measurement issues surrounding 
particular outcomes, following the order in which those outcomes appear in Exhibit 1 . 

Transition to, and Persistence in, Postsecondary Education'* 

In some states, it will be difficult to track tech prep students from the secondary to the 
postsecondary level. Because of concerns about compliance with the Family Educational Rights 
and Privacy Act (FERPA), many postsecondary institutions will not provide the data that local 
consortia need to monitor this outcome, or to help them determine how many tech prep students 



Additional outcome identified by the group. 
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enter postsecondary schools without the need for remediation.^ States will have to deal with this 
issue, however, in order to comply with the Perkins HI performance requirements. 

Improved Awareness of Career Options^ 

As a result of institutional changes, one group member observed, tech prep students are 
often more career-oriented, and have a stronger sense of self-efficacy, than other students. 
Although these outcomes would be difficult to measure, a national evaluation could conceivably 
include measures of self-esteem and locus of control for a sample of tech prep students and a 
comparison group. Alternatively, it might be possible to simply measure how much access 
students had to career development activities. 

Achievement of Employability SkillsA/Vork Readiness 

While this is an important outcome, it will be difficult to document since few states use 
appropriate assessments. This outcome does not pertain to the integration of instruction, as 
suggested by the exhibit, but only to the use of work-based or worksite learning. (Integrated 
instruction might be linked, instead, with improvements in academic skills.) 

Placement in Program-Related Career and 

Retention of, and Advancement in, Employment in Program-Related Career 

These outcomes are designed to monitor the effect of Perkins Ill’s requirement that tech 
prep programs “provide technical preparation in a career field.” The legislation does not specify, 
however, what level of competence students must achieve. Further, “technical preparation” can 
be difficult to define. 

Several group members reported that their states (including Illinois and Arizona) either 
were currently attempting, or had previously attempted — with considerable difficulty — to 
monitor students’ entry into program-related careers. They recommended, and other group 
members concurred, that a national evaluation should not attempt to track placement, retention. 



Please see “Other Design and Data Collection Issues” for a discussion of technical assistance needs in this 



area. 
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or advancement m program-related careers. Another reason for deleting this phrase is that 
federal policy is now promoting the development of broad career clusters; consequently, it is not 
desirable to select outcomes that focus too narrowly on specific occupations. Finally, because so 
many changes are taking place in the world of work, students may have good reasons to change 
their career goals. 

In the current economy, employers may hire tech prep students as soon as they have 
entry-level skills, even if they have not completed their course of study. These individuals often 
continue to attend school part-time and may eventually complete the program; the most 
immediate benefits for them, however, will be in the form of employment, job retention, and 
earnings. Perhaps, participants suggested, a combination of educational and employment data 
would help to “tell the story” of the program’s impact. 

Group members agreed that entering employment immediately after high school should 
not be counted as a successful outcome. Perkins HI, they noted, refers to only one option for 
employment after high school: placement in military service.^ 

Defining a Tech Prep Student 

Participants noted that a single definition of a tech prep student would have to be 
somewhat vague in order to allow room for interpretation by individual states and consortia. 

This lack of specificity, however, creates difficulties for evaluators. 

In this section, we describe group members’ concerns pertaining to specific aspects of the 
definition developed by NATPL, which is shown in Exhibit 2. We also present general themes 
concerning the definition of a tech prep student that arose from the group’s discussions. 

According to focus group members who are also involved in NATPL, the association 
does not endorse the definition, but merely offers it for discussion. It is based on an 
identification of common elements in definitions submitted by about 20 states that responded to a 
request for information (including some that did not have, or were in the process of developing, 
definitions). In a few cases, different entities in the same state submitted conflicting information. 




7 



Perkins II also referred to placement in a two-year apprenticeship program. 
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Concerns Pertaining to Specific Aspects of the Definition 

Intent to pursue. Only one participant indicated that her state (Maryland) relied on 
“intent” to identify tech prep students: in this state, 10“' grade students indicate their intent to 
participate in a tech prep program by signing a five-year education plan. Maryland has found 
that — while students may change from one area within a career cluster to another — they do not 
usually change from one cluster to another. Other group members, however, believed that 
secondary students frequently changed course sequences; consequently, they expressed concern 
about the feasibility of relying on intent to identify tech prep students. 

Enrolled in courses within a recognized tech prep education plan. Focus group 
members observed that the phrase “recognized tech prep education plan” might mean something 
in some states, but not in others. About half reported that their states required students to have 
some sort of education plan: South Carolina, Illinois, and West Virginia do, while California, 
Washington, and Arizona do not. The existence of such a plan depends to some extent upon the 
availability of counseling and guidance services: if the state does not require the development of 
education plans, or if insufficient numbers of counselors are available, some students may “fall 
through the cracks.” 

Asked whether ED should recommend that states require such a plan, one respondent 
remarked that states might welcome guidance in this area. Such a recommendation might be 
controversial, however, if federal guidelines concerning education plans also established 
minimum standards for academic and vocational coursework. 

May allow the student to earn postsecondary credit while in secondary school 
(secondary definition) and The student may have transferred in college credit earned in the 
secondary school (postsecondary definition). Transferring credit from high school to college, 
one participant observed, is not sufficient to define a tech prep student. In some states, local 
districts have numerous articulation agreements; sometimes, however, relatively few students 
take advantage of them. Consequently, we should remove the above phrases from the secondary 
and postsecondary definitions. 
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General Themes 

The following general themes concerning the definition of a tech prep student also arose 
from the group's discussions. 

Options for identifying tech prep students. Most group members indicated that their 
states and local consortia could identify tech prep students, although they use a variety of 
definitions. Another possible approach for a national evaluation would be to identify tech prep 
students as they complete the secondary portion of the program. This approach would allow for 
the application of a more uniform definition; e.g., students could be identified based on course- 
taking patterns. The disadvantage of this strategy is that it would make it impossible to examine 
the impact of tech prep on student retention. Further, some might argue that a tech prep student 
who completes the secondary portion of the program is only a concentrator, rather than a 
completer, since the program encompasses both the secondary and postsecondary levels. Instead 
of identifying students retrospectively, one participant suggested, a national evaluation that 
included a longitudinal component could identify them based on intent early in their high school 
careers, and then track individuals over time to see whether they actually completed the program. 

Many students who complete only part of a tech prep program, focus group members 
pointed out, still benefit. Evaluators might be able to document these benefits by studying 
outcomes not only for participants and completers, but also for tech prep concentrators. Students 
who drop in and out of the program over time may also benefit; however, states may have limited 
ability to track these individuals. 

Other Design and Data Collection Issues 

Under this topic, participants discussed several issues, including: (1) the need to include 
input and contextual variables in the design of a national evaluation; (2) how a national 
evaluation should be coordinated with Perkins III requirements; and (3) state and local needs for 
technical assistance. 
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Input and Contextual Variables 

Tech prep programs are being implemented in an economic context that differs from the 
environment in which the program was first conceived. Most community college students now 
work while attending school; as a result, they often take longer to complete their studies. With 
unemployment rates low, job seekers may be able to obtain employment with fewer educational 
credentials than were required in the past. Finally, because individuals who are qualified to teach 
technical courses can frequently obtain more lucrative jobs in business and industry, 
postsecondary institutions may have difficulty in attracting and retaining qualified instructors. 

According to focus group members, to assist evaluators in interpreting outcome data, a 
national evaluation should examine — ^perhaps through case studies — the extent to which states 
and local programs have implemented particular program elements, as well as the characteristics 
of the communities in which tech prep programs operate. Researchers would also benefit from 
access to information collected by other education reform initiatives, including school-to-work 
and High Schools That Work (HSTW). 

When school-to-work funds became available, one participant observed, some tech prep 
programs “literally changed their names” because of similarities between the two initiatives; in 
other cases, they continued to operate as they had in the past. Similarly, the discontinuation of 
federal funding for school-to-work programs in 2001 is also likely to impact tech prep programs. 
Evaluations of the school-to-work program might also provide data on tech prep students, as 
would data from HSTW evaluations. West Virginia, for example, obtains science, math, and 
reading scores for its tech prep students from the latter source. 

Coordination with Perkins III Requirements 

An OVAE representative identified a number of discrepancies between the Perkins III 
core indicators and the outcomes shown in Exhibit 1, including the following: 

• Exhibit 1 refers to.- 

► Improvements in academic skills, while Perkins refers to skill attainment. 

► Advancement in career, while Perkins III looks only at placement and 
retention. 
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► Workplace skills, while Perkins in refers to technical skills. 

► Attainment of associate degrees, baccalaureate degrees, and 
postsecondary certificates and credentials. The Perkins m core indicators 
will monitor attainment of postsecondary degrees and credentials, perhaps 
without distinguishing what type of degree or credential the individual 
earned. Evaluators responsible for a study of the tech prep program should 
consider whether the core indicators will track students for a sufficient 
length of time to document their entry into four-year programs. 

• Perkins III does not refer to: 

► Completion of an articulated program; 

► Employability skills and work readiness; or 

► Program-related careers. 

Exhibit 6 provides a side-by-side comparison of the Perkins m core indicators and the outcomes 
discussed by the focus group. 

States must provide core indicator data for all vocational education students, including a 
separate report for tech prep students. They may, however, use a variety of measurement 
approaches; for example, to measure attainment of academic skills at the secondary level, states 
may employ their own or national standardized assessment systems, grade point average, 
program completion, high school graduation, or another specified measure. Participants agreed 
that it would be desirable to align data collection requirements for Perkins HI and a national tech 
prep evaluation to the extent possible, in order to permit comparison of tech prep students and 
vocational education students in general. In a national evaluation, however, all states could be 
required to employ a single measure. 

Perhaps more importantly, an evaluation would allow researchers to supplement the core 
indicator data with additional information that might be more pertinent to the tech prep program. 
For example, states may measure attainment of academic proficiencies by using 10“’ grade 
assessments that monitor achievement of minimum standards. Tech prep programs, however, 
emphasize transition to postsecondary education, often through activities that take place during 
the last two years of high school. As a result, a more appropriate assessment for tech prep 
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Exhibit 6 

Comparison of Core Indicators and Exhibit 1 Outcomes* 



Perkins III Core Indicators 


Exhibit 1 Outcomes 


Student attainment of challenging state- 
established academic, and vocational and 
technical, skill proficiencies 


• Improvements in academic skills 

• Improvements in workplace skills 


Student attainment of a secondary school 
diploma or its recognized equivalent, a 
proficiency credential in conjunction with a 
secondary diploma, or a postsecondary 
degree or credential 


• Attainment of associate degree in career 
field 

• Attainment of baccalaureate degree in 
career field 

• Attainment of postsecondary certificate in 
career field 

• Attainment ol postsecondary credentials 
in career field 


Placement in, retention in, and completion of 
postsecondary education or advanced training, 
placement in military service, or placement or 
retention in employment 


• Completion of articulated 
secondary/postsecondary program 

• Transition to baccalaureate program in 
career field 

• Transition to additional postsecondary 
training in career field 


Student participation in and completion of 
vocational and technical education programs 
that lead to nontraditional training and 
employment 




— 


• Placement, retention, and advancement in 
program-related career 


— 


• Achievement of employability skills/work 
readiness 



* Emphasis added. 
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students might be one that is administered at the 12^*' grade level, and that monitors attainment of 
skills necessary for entry into college (e.g., attainment of requirements for entry into state- 
supported colleges). Completion of an articulated program and achievement of employability 
skills/work readiness might also be good measures to include for tech prep. 

Technical Assistance Needs 

States would benefit from technical assistance to help them collect comparable data for 
both tech prep and other vocational education students. Perhaps, participants suggested, OVAE 
should consider a pilot test similar to the one that it is conducting for the Perkins III core 
indicators. 

As one participant suggested, .another important role for OVAE would be to help address 
educational institutions’ concerns about compliance with FERPA. One focus group member 
indicated that her state had recently received federal guidance in this area that caused some 
community colleges to refuse to release the names of students who were enrolled in secondary 
tech prep programs. One way to address these concerns, according to an OVAE representative, 
is to obtain a wai ver from each student (and from parents, in the case of minors). 

ED should also educate state officials about the importance of conducting their own tech 
prep evaluations. In anticipation of the Mathematica study, one participant noted, some states 
postponed their own activities; yet that study had only limited utility at the state and local levels. 
Federal officials should indicate that a national study will build on, and not replicate, state 
efforts, and encourage states to share evaluation data with them. One way to promote the 
importance of state activities would be to distribute the focus group report to state directors of 
vocational technical education and state tech prep coordinators. 

Focus group members identified two additional areas in which OVAE might be able to 
assist state and local tech prep programs: (1) by helping states build their capacity to conduct 
research; and (2) by encouraging collaboration between tech prep and other educational 
initiatives such as school-to-work, HSTW, and New American High Schools. In the latter area, 
states and local programs would benefit from case studies that showed where the initiatives are 
coordinated, and are part of a true “system.” 
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Group 3 



Group 3 included three state tech prep coordinators and an equal number of individuals 
selected for their expertise in the conduct of national- or state-level evaluations of tech prep 
programs. One participant represented a national organization, and one a local consortium. 
Among the major points of the group’s discussions were: 

• The possible addition of other outcomes, including transition to postsecondary 
education without the need for remediation, and earnings data; 

• The need to conduct an evaluation that would provide both data to support 
funding requests and information useful for program improvement; 

• Variation among state and local definitions of a tech prep student; and 

• Appropriate timing for a national tech prep evaluation. 

Outcomes 

Focus group members agreed that — although not every tech prep consortium can 
implement all elements of the program— all of the outcomes shown in Exhibit 1 could help to 
reflect the overall impact of tech prep. Among the most critical are: improvements in academic 
skills, placement in program-related career, achievement of employability skills/work readiness 
(including “soft” and SCANS skills), and improvements in workplace skills (including technical 
skills). 

Additional outcomes identified by group members include: 

• Transition to postsecondary education, which is an important emphasis of the 
tech prep program. 

• Entry into postsecondary education without the need for remediation. An 
evaluation should investigate the question of whether — as proponents of tech prep 
believe — participants are well prepared for postsecondary education. One way to 
do this would be to compare their performance on postsecondary placement tests 
to that of other students (as Florida has recently done). However, because 
individual institutions and states employ a wide variety of placement tests, entry 
into postsecondary education without the need for remediation might be a better 
indicator. 
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• Earnings. Congress has made it clear, one participant observed, that it not only 
expects vocational programs to improve academic achievement and college-going 
fates, but also to increase earnings. Consequently, a national tech prep evaluation 
should collect information on students’ initial and later earnings. Earnings data 
might also improve the “marketability” of the program. 

• Employment and earnings in combination with postsecondary education. The 
technical skills that many tech prep students (especially those in fields such as 
Information Technology) acquire allow them to obtain high-paying jobs while 
they are still enrolled in community colleges. This is an important outcome, 
especially since students’ earnings enable them to complete their postsecondary 
education. Unemployment Insurance wage data might help in monitoring this 
outcome, although some students are self-employed entrepreneurs. 

Fociis group members noted that a study of the tech prep program should provide both 
student outcome data to support funding requests at the federal, state, and local levels and 
contextual information that administrators and practitioners can use for program improvement. 
They disagreed, however, about what type of study, or studies, would best accomplish these dual 
purposes. 

One option would be to conduct an outcome evaluation that also included in-depth case 
studies at about a dozen sites. Still, one participant argued, it would be impossible to say that 
individual program elements produced particular student outcomes. There is, however, some 
evidence to suggest that certain combinations of program elements are more effective than 
others. To examine the link between student outcomes and program elements, OVAE could 
design a demonstration program involving consortia that emphasized different approaches to tech 
prep implementation. In such a study, students could be randomly assigned to tech prep or other 
programs at each site. 

In order to identify appropriate student outcomes, researchers must develop hypotheses 
about how particular program elements impact students. Individual consortia may not be able to 
implement all elements of the program, especially given current funding levels; and, in the 
absence of all program elements, it may not be appropriate to expect certain student outcomes. 

For example, a national evaluation that relies on a random sample of local programs will 
undoubtedly include some consortia that use their tech prep funds solely for staff development. 
Since students at those consortia may not achieve the same academic outcomes as those in more 
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comprehensive programs, it is important to have contextual information that facilitates accurate 
interpretation of data for those sites. 



In this section, we present the group’s concerns about measurement issues surrounding 
particular outcomes, following the order in which those outcomes appear in Exhibit 1. We also 
describe one general concern, pertaining to the timing of a national evaluation, that emerged from 
the focus group’s discussions. 

Measurement Issues for Particular Outcomes 

Completion of articulated secondary/postsecondary program. A national evaluation 
may not be able to track all students who eventually complete tech prep programs. Although 
many students “drop in and out” of postsecondary education, only those who are enrolled at a 
particular point in time will be counted. Students who move from one college to another may be 
hard to locate. Finally, because many individuals require more than the scheduled length of time 
to complete their studies, evaluators may need to track students for an extended period of time. 

Most group members indicated that monitoring entry into postsecondary education was 
not a simple process in their states. Secondary and postsecondary systems may not be linked, or 
may provide only aggregate data. At the secondary level, data systems often do not include 
social security numbers. 

Two participants reported that their states (Arizona and Ohio) provide financial 
incentives to local consortia that monitor progress from the secondary to the postsecondary 
level.* Tech prep persormel in Arizona have tracked students by hand in the past, but are now 
investigating a computer program that would facilitate their work; in Ohio, tech prep consortia 
use a web-based system. Local programs in the latter state receive $250 for each tech prep 
student who enters postsecondary education, with a $250 bonus if the student does not need 



^Arizona’s incentive also applies to students who enter employment or the military within two years after 
high school graduation. 



Measurement 
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remediation. The state is also undertaking a study to investigate the reasons why some tech prep 
students do not go on to college. 

Placement in program-related career and retention of, and advancement in, 
employment in program-related career. One participant expressed concern about attempting 
to monitor placement, retention, and advancement in program-related careers. Individuals may 
have several “careers” during their lifetimes; further, a nonrelated career may sometimes 
represent a better career choice, or indicate that students’ skills are transferable. Another group 
member suggested that an evaluation could study placement in both program-related and 
unrelated careers without making a judgment that one was better than the other.’ This 
information might be useful for program improvement, since it could impact counseling 
practices. 

Improvements in academic and workplace skills. At the postsecondary level, 
evaluators could employ Grade Point Average (GPA) as a measure of academic skills. At the 
secondary level, however, improvements in academic skills might be more difficult to measure. 
While most states test at the 10*'’ grade level, many tech prep activities do not take place until the 
last two years of high school. Consequently, one participant argued, evaluators responsible for a 
national study of the tech prep program could not make use of existing assessment data in most 
states, and would have to administer their own pre- and post-tests. These data could be quite 
informative; test administration, however, would consume a considerable amount of resources. 

Several other participants, however, disagreed strongly with this argument. “Only at the 
point that our tech prep initiative became a part of our statewide accountability (system),” North 
Carolina’s state tech prep coordinator pointed out, “did my phones start ringing.” Because tech 
prep completers in the state had posted above-average scores on the HSTW National Assessment 
of Educational Progress mathematics and reading assessments, principals in the state believed 
that the program could help improve test scores. Similarly, an individual from New York 
reported that interest in tech prep “. . .is absolutely driven by superintendents saying tech prep 
helped (students pass the state’s Regents exams). Otherwise, you’re out of the game.” 



’Consequently, outcomes should include not only “placement in program-related career,” but also 
“placement in career.” 
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Improvements in workplace skills are an important outcome, but one that is difficult to 
measure. While some states have appropriate assessments (e.g., Ohio uses Work Keys), others 
do not. The best way to measure this outcome might be through employer surveys, which not all 
states currently do at both the secondary and postsecondary levels. Survey results, however, 
would need to be linked to individual students, rather than aggregate data. In a demonstration 
program, researchers could administer an appropriate assessment to a sample of 9*** grade 
students, and then compare the performance of 12"’ grade tech prep students with that of other 
seniors. 

Timing of a National Evaluation 

A national tech prep evaluation, one focus group member argued, should be completed 
before the next reauthorization of the Perkins legislation. Student outcome data could help to 
ensure that the legislation governing vocational education remains separate fi-om that of other 
programs: a major concern among state directors of vocational education. 

Evaluation results, however, will not be available by that time (2002). State performance 
reports and the National Assessment of Vocational Education (NAVE) may provide useful 
information; or, OVAE may wish to consider whether it can design a study that provides interim 
products that would be useful in the reauthorization process. In any event, the evaluation is 
designed not only for legislative, but also for program improvement, purposes. 

Defining a Tech Prep Student 

Several members of the focus group provided their states’ definitions of a tech prep 
student for discussion; these are shown in Exhibit 7. Some believed that the definition of a tech 
prep student developed by NATPL would be applicable in their states; however, participants 
mentioned a number of concerns about specific aspects. In this section, we describe those 
concerns. We also present general themes concerning the definition of a tech prep student that 
arose from the group’s discussions. 



'®For a discussion of NAVE activities pertaining to tech prep, please see “Other Design and Data 
Collection Issues," 
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Exhibit 7 

State Definitions of a Tech Prep Student 



Arizona. A tech prep student is any student enrolled in one or more occupationally based secondary 
and/or postsecondary programs in an identified Career Pathway. Students must be enrolled in one of 43 
state-approved vocational areas. 

Florida. A student enrolled in an articulated, sequential program of study, at grade level or above, by 
grade 1 1, in mathematics, science, and communications, including a technical component, which leads to 
a minimum of a two-year postsecondary certificate or degree, and/or apprenticeship program. 



New York. A tech prep student: 



a. Receives academic content through applied learning; 



. b. Completes a sequence of two or more standards-based, career-oriented courses that 
provide technical skills; 

c. Is enabled to be employed in skilled entry level jobs and completes the postsecondary 
portion of a secondary/postsecondary program; 



d. If a postsecondary student, has completed the secondary portion of 

secondary/postsecondary program and has enrolled in an articulated, state registered 
postsecondary tech prep program; and 



e. Is a completer of a two-year associate degree program, or a two-year certificate program, 
or a two-year apprenticeship program that follows secondary tech prep instruction. 



North Carolina. North Carolina identifies tech prep completers through a computerized analysis of 
course-taking patterns. Requirements for the state’s College Tech Prep Course of Study include: (1) four 
units of credits in career/technical courses, including an advanced course; and (2) three credits of 
mathematics, including either Algebra 1, Algebra 2, and Geometry; Algebra 1, Technical Math 1, and 
Technical Math 2; or Integrated Mathematics 1, 2, and 3. Mathematics requirements are very similar to 
those for students in the state’s College Prep Course of Study. 



Ohio. Students in state-approved tech prep programs in grades 11 or 12, and those students who have 
matriculated from a high school tech prep program to one operating at the college level. 
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Concerns Pertaining to Specific Aspects of the Definition 

Intent to pursue. None of the definitions used by participants’ states depended upon a 
student indicating an intent to pursue a tech prep program. In fact, two (New York and North 
Carolina) do not identify tech prep students until they have completed secondary school. This 
approach, one participant suggested, may be preferable, since students’ intentions often change 
(particularly at the 1 T'’ grade level, which seems to be a “defining moment” in their high school 
careers). 

Whether a national evaluation depends on a definition that includes intent, or identifies 
tech prep students retrospectively (i.e., after they complete the secondary portion of the program) 
has major implications for the design of the study. Identifying students on the basis of intent 
would allow researchers to administer assessments as the students progressed through high 
school. The disadvantage of this approach is that such a design might hold states and local 
programs accountable for students who did not complete the program. 

If, on the other hand, an evaluation identifies students retrospectively, it cannot track their 
progress over time in secondary school. Further, students would be identified in such a way that 
they were automatically likely to be the most successful academically (i.e., those with the highest 
academic ability would be most likely to complete the program). To address these issues, 
researchers might need to select appropriate comparison groups, while controlling for academic 
ability. 

Carried out under a written articulation agreement. Participants noted that the term 
“written articulation agreement” could refer to either a broad statewide agreement or a 
specialized local initiative. 

May allow the student to earn postsecondary credit while in secondary school 
(secondary definition) and The student may have transferred in college credit earned in the 
secondary school (postsecondary definition). In some places, one participant noted, individuals 
who earn postsecondary credit while in secondary school are Advanced Placement students 
enrolled in dual credit programs. Rather than focusing on postsecondary credit, she suggested, a 
definition should emphasize participation in an articulated program. 
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General Themes 

The following general themes concerning the definition of a tech prep student also arose 
from the group's discussions. 

Desirability of identifying tech prep students. Through a representative of their 
association, some state directors of vocational education suggested that, while it might be 
necessary to identify tech prep students in order to study the impact of the program, it was not 
necessary to “label” individuals as tech prep students. Tech prep is a part of vocational 
education, and in some cases can be difficult to distinguish from other forms of vocational 
education. 

Variation in local deflnitions. Even where state definitions do not rely on student 
intent, local consortia may depend partially upon intent to identify tech prep students. In Florida, 
for example, only about 20 percent of the state’s 28 local consortia have adopted the statewide 
definition. In other cases, local definitions may depend on a formal application process or on a 
student’s being enrolled in particular courses. 

Other Design and Data Collection Issues 

Under this topic, participants discussed several issues, including: (1) the need to include 
input and contextual variables in the design of a national evaluation; and (2) how an evaluation 
of the tech prep program should be coordinated with other national activities, including the 
NAVE and data collection requirements for the Perkins III core indicators. 

Input and Contextual Variables 

Among the most important contextual variables for a national evaluation to consider 
would be the level of state and local funding for tech prep programs. Perkins funds, one 
participant noted, are discretionary and thus can be extremely valuable at the local level. 

Program success, however, may depend partially upon the level of state and local cash and in- 
kind support leveraged by the federal funds. Evaluators could also examine the way in which 
tech prep funds are allocated at the statewide level. 
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An evaluation might also collect information on the statewide context within which tech 
prep operates; e.g., whether the state has its own legislation governing the program and where 
within state government the program is housed. Researchers could also investigate issues 
concerned with leadership, such as the level of support that tech prep receives from the state 
director of vocational and technical education and how activities funded by the basic state grant 
and tech prep are coordinated. 

Coordination with Other Nationai Activities 

Evaluators responsible for a national tech prep evaluation may want to consider how their 
activities complement those of the NAVE, which is to report to Congress by 2002. As part of 
this assessment, ED will receive extensive data for all public school students (including 
participants in tech prep, vocational education, and other programs) in three states. This 
information — which will include transcript data and assessment data, secondary school records, 
two-year and four-year college records, and UI wage data — will allow researchers to examine 
program effectiveness using a variety of outcome measures. ED personnel will also conduct site 
visits to the three states and to several local sites within each of the states, in order to obtain 
contextual information. Researchers do not yet know whether the three states have statewide 
definitions of a tech prep student. 

The focus group also discussed the need to align evaluation requirements with data 
collection requirements for the Perkins EQ core indicators. Although the legislation requires 
states to monitor the same outcomes for tech prep and other vocational education students, some 
measures may be more appropriate for the tech prep program than others (and thus more likely to 
be useful for program improvement). North Carolina, for example, uses a community college 
placement exam (rather than state academic assessments) to document academic achievement. 
Some educators, however, might not be comfortable with the notion of “rank-ordering” outcomes 
with preparation for college as the most desirable. 
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Appendix A: Focus Group Participants 



Ms. Linda Avelar, Dean, Business Division, Skyline College, San Bruno, CA 

Ms. Fran Beauman, Division Administrator, Workforce Preparation Partnerships, Illinois State 
Board of Education, Springfield, EL 

Ms. Bonnie Bensonhaver, Sinclair Community College, Dayton, OH 

Mr. David Bond, Vice President, Education/Employer Partnerships, Center for Occupational 
Research and Development, Waco, TX 

Ms. Debra Bragg, Associate Professor, University of Illinois, Champaign, EL 

Dr. Carrie Brown, Director, Tech Prep Statewide Leadership and Evaluation Project, Beaumont, 
TX 

Dr. Jerry Ciesla, Senior Partner, MGT of America, Inc., Tallahassee, FL 

Ms. Diana Crowley, Tech Prep Coordinator, Rhode Island Department of Education, Providence, 

RI 

Ms. Nancy Dillon, State Board of Directors (State Tech Prep Coordinator), Community Colleges 
of Arizona, Phoenix, AZ 

Ms. Kathy D* Antoni, West Virginia Joint Commission for Voc Tech (State Tech Prep 
Coordinator), Charleston, WV 

Df. Margaret Ellibee, Field Service Coordinator, Oklahoma Department of Vocational and 
Technical Education, Stillwater, OK 

Ms. Kimberly Greene, Executive Director, National Association of State Directors of Vocational 
Technical Education, Washington, DC 

Dr. Frank Hammons, Associate Professor and Director, Institute for Workforce 
Competitiveness, Florida International University, Miami, FL 

Ms. Arlene McCollum, Director, Tech Prep Consortium, Green River Community College, 
Auburn, WA 

Dr. Bernie Mclnerney, State Tech Prep Coordinator, New York State Education Department, 
Albany, NY 

Dr. Jim McKenney, Director of Economic Development, American Association of Community 
Colleges, Washington, DC 
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Ms. Debbie Mills, Tech Prep Director, Danville Area Community College, Danville, VA" 

Ms. Kathy Oliver, Assistant State Superintendent (State Director of Vocational Technical 
Education), Maryland State Department of Education, Baltimore, MD 

Mr. Wofford O’Sullivan, Education Associate, South Carolina Department of Education 
(representing State Director of Vocational Technical Education), Columbia, SC 

Mr. Gerald Pumphrey, Vice President for Instruction, Guilford Technical Community College, 
Jamestown, NC 

Dr. Marsha Silverberg, Planning and Evaluation Service, U.S. Department of Education (on 
loan from Mathematica Policy Research) 

Mr. Kenneth Smith, State Tech Prep Coordinator, Department of Public Instruction, Raleigh, 

NC 

Ms. Connie Spohn, Coordinator, Greater Capital District Tech Prep Consortium, Albany, NY, 
and Director, Two-Year College Development Center, University at Albany 



Unable to Attend: 

Dr. Joseph Crossen, Curriculum Development, Delaware Department of Education (State 
Director of Vocational Technical Education), Dover, DE 



OVAE Participants 



Ms. Gisela Harkin, Division of Vocational-Technical Education, Office of Vocational and Adult 
Education, U.S. Department of Education 

Ms. Laura Messenger, Division of National Programs, Office of Vocational and Adult 
Education, U.S. Department of Education 




"After participant selection, Ms. Mills accepted a position with CORD. 
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Other OVAE Observers 



Ms. Nancy Brooks, Division of National Programs, Office of Vocational and Adult Education, 
U.S. Department of Education 

Ms. Cynthia Brown, Policy Analysis Staff, Office of Vocational and Adult Education, U.S. 
Department of Education 

Mr. Ron Castaldi, Director, Division of Vocational-Technical Education, Office of Vocational 
and Adult Education, U.S. Department of Education 

Mr. Maury James, Chief, State Coordination Branch, Division of Vocational-Technical 

Education, Office of Vocational and Adult Education, U.S. Department of Education 

Ms. Vickie Schray, Chief, Program Analysis Branch, Division of Vocational and Technical 
Education, Office of Vocational and Adult Education, U.S. Department of Education 
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